Skip to content

Remove opendal dependency from QEMU driver#535

Open
ambient-code[bot] wants to merge 5 commits intomainfrom
ambient-fix/441-remove-opendal-qemu
Open

Remove opendal dependency from QEMU driver#535
ambient-code[bot] wants to merge 5 commits intomainfrom
ambient-fix/441-remove-opendal-qemu

Conversation

@ambient-code
Copy link
Copy Markdown
Contributor

@ambient-code ambient-code bot commented Apr 9, 2026

Summary

  • Move FlasherInterface ABC to core jumpstarter package (jumpstarter.driver.flasher) — it's a pure abstract class with no opendal imports
  • Create a simplified FlasherClient in core (jumpstarter.client.flasher) that handles local files via resource_async and HTTP URLs via PresignedRequestResource, without requiring the opendal library
  • Update QEMU driver to import FlasherInterface from jumpstarter.driver instead of jumpstarter_driver_opendal.driver
  • Remove jumpstarter-driver-opendal from QEMU's pyproject.toml dependencies
  • Update test to use direct HTTP URLs instead of opendal.Operator

Backward Compatibility

  • The opendal package's own FlasherInterface is unchanged — drivers that depend on opendal (esp32, pi-pico, dutlink, sdwire) continue to work without modification
  • The core FlasherInterface.client() points to jumpstarter.client.flasher.FlasherClient (simplified, no opendal)
  • The opendal FlasherInterface.client() continues pointing to jumpstarter_driver_opendal.client.FlasherClient (full opendal support)

Test plan

  • All 7 QEMU unit tests pass (resize, set_disk_size, set_memory_size)
  • Integration test (test_driver_qemu) fails only due to missing qemu-img binary in CI environment (pre-existing)
  • Linting passes on all changed files
  • Verify other drivers (esp32, pi-pico, dutlink, sdwire) still import correctly from opendal

Related #441

🤖 Generated with Claude Code

@netlify
Copy link
Copy Markdown

netlify bot commented Apr 9, 2026

Deploy Preview for jumpstarter-docs failed. Why did it fail? →

Name Link
🔨 Latest commit bffbd65
🔍 Latest deploy log https://app.netlify.com/projects/jumpstarter-docs/deploys/69d917eb17b9810009854c5b

@coderabbitai
Copy link
Copy Markdown
Contributor

coderabbitai bot commented Apr 9, 2026

Important

Review skipped

Bot user detected.

To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 8a7e580c-3aa3-4b37-82b4-18b24bc013f4

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch ambient-fix/441-remove-opendal-qemu

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@ambient-code
Copy link
Copy Markdown
Contributor Author

ambient-code bot commented Apr 9, 2026

Fixed the CI failures by adding click>=8.1.7.2 to the jumpstarter package dependencies.

The new jumpstarter/client/flasher.py file imports click for its CLI functionality, but this dependency was missing from the package's pyproject.toml. This caused ModuleNotFoundError: No module named 'click' errors when tests tried to import modules that depend on the flasher client.

The version requirement >=8.1.7.2 matches the convention used in other jumpstarter driver packages.

@@ -0,0 +1,286 @@
"""
Simplified FlasherClient that does not depend on opendal.
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it's reasonable to start accumulating common driver interfaces in the core jumpstarter, as, for example, this one is broadly used around Jumpstarter, and lived on opendal initially.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed! The FlasherInterface ABC is purely abstract with no external dependencies, so it makes sense for it to live in core. This way any driver can implement it without pulling in opendal as a transitive dependency.

@mangelajo mangelajo enabled auto-merge (squash) April 10, 2026 09:35
@mangelajo mangelajo disabled auto-merge April 10, 2026 09:39
local_path, url = _parse_path(path)

if url is not None:
with _http_url_adapter(client=self, url=url, mode="wb") as handle:
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are we testing all paths in here: https://github.com/jumpstarter-dev/jumpstarter/blob/main/python/packages/jumpstarter-driver-qemu/jumpstarter_driver_qemu/driver_test.py#L79 ?

I see it uses the Operator, can we make sure that we take that and interpret that for compatibility purposes without really relying/using opendal?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good question. The current test covers two paths:

  1. Local file path (cached_image.resolve()) -- this exercises the _local_file_adapter code path in the new FlasherClient, streaming the file via resource_async.
  2. HTTP URL (the Fedora download URL) -- this was previously using Operator("http", endpoint=...) and is now a direct https:// URL, which exercises the _http_url_adapter code path using PresignedRequestResource.

Both paths in the new FlasherClient are covered by the integration test.

Regarding backward compatibility with opendal.Operator: the opendal package's FlasherClient (in jumpstarter_driver_opendal.client) is unchanged and still accepts the operator= parameter. Drivers that depend on jumpstarter-driver-opendal (esp32, pi-pico, dutlink, sdwire) continue to use that client via their own FlasherInterface.client() classmethod which points to the opendal version.

The QEMU driver now uses the core FlasherInterface which points to the simplified FlasherClient -- this one intentionally does not accept an operator= parameter since the goal is to avoid the opendal dependency entirely. The Operator usage in the old test was only needed because the old flasher client required it for HTTP downloads; the new client handles HTTP URLs natively.

So the API surface for QEMU users changes from:

# Old (required opendal)
qemu.flasher.flash("path/on/server", operator=Operator("http", endpoint="https://example.com"))

# New (no opendal needed)
qemu.flasher.flash("https://example.com/path/on/server")

The new API is simpler and doesn't break any existing QEMU usage since the operator= parameter was only used in this test, not in production QEMU workflows.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, but if that test is there, is not executing. Should we make sure to have a test that executes the full url download?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You're right -- when the cached image exists (which it does in CI), the HTTP URL branch is never exercised.

I'll add a dedicated unit test for the _http_url_adapter and _parse_path helpers in the core jumpstarter package to make sure the URL code path is covered without needing a full Fedora image download. This will verify that:

  1. _parse_path correctly distinguishes local paths from HTTP/HTTPS URLs
  2. _http_url_adapter produces a proper PresignedRequestResource for HTTP URLs
  3. The FlasherClient._flash_single method routes HTTP URLs through the presigned request path

For a full end-to-end HTTP download test, that would involve downloading a large image in CI which is expensive. I think a unit test covering the routing logic is the right trade-off here. Does that work for you?

@ambient-code
Copy link
Copy Markdown
Contributor Author

ambient-code bot commented Apr 10, 2026

Added unit tests for FlasherClient URL routing in 6bacdd62 to address @mangelajo's feedback about test coverage for the HTTP URL code path.

The new tests in flasher_test.py cover:

  • _parse_path correctly distinguishing local paths from HTTP/HTTPS URLs
  • _http_url_adapter producing correct PresignedRequestResource for read (GET) and write (PUT) modes
  • FlasherClient._flash_single routing HTTP URLs through the presigned request path (not the local file adapter)
  • FlasherClient.dump routing HTTP URLs correctly

This ensures the URL download logic is exercised in CI without needing to download a full Fedora image.

@ambient-code
Copy link
Copy Markdown
Contributor Author

ambient-code bot commented Apr 10, 2026

Fixed the lint-python CI failure in 6c8eee9 — removed two unused imports in flasher_test.py that ruff flagged as F401:

  • pathlib.Path (line 3)
  • jumpstarter.common.resources.PresignedRequestResource (line 56)

The netlify deploy preview failure is unrelated to this PR's changes.

@mangelajo
Copy link
Copy Markdown
Member

mangelajo commented Apr 10, 2026

Rebase the pr please.

@ambient-code ambient-code bot force-pushed the ambient-fix/441-remove-opendal-qemu branch from 6c8eee9 to bffbd65 Compare April 10, 2026 15:31
@ambient-code
Copy link
Copy Markdown
Contributor Author

ambient-code bot commented Apr 10, 2026

Rebased onto main (d7a65e0). The branch now has a clean history on top of the latest main.

@ambient-code
Copy link
Copy Markdown
Contributor Author

ambient-code bot commented Apr 12, 2026

Rebase complete. The branch ambient-fix/441-remove-opendal-qemu is already up to date with main (merge base is the current tip of main at d7a65e0). No conflicts were encountered.

@raballew raballew self-requested a review April 13, 2026 17:10
def flash(self, source, target: str | None = None): ...

@abstractmethod
def dump(self, target, partition: str | None = None): ...
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[MEDIUM] Parallel FlasherInterface ABCs with no inheritance relationship

There are now two identical FlasherInterface ABCs -- this one in core and the one in jumpstarter-driver-opendal -- with no inheritance link between them. Both define flash() and dump() with the same signatures but different client() return values. Any future contract change must be manually synchronized.

Since this is phase 1 of the incremental migration (#441), consider having the opendal FlasherInterface inherit from this core one in a follow-up, or plan to remove the opendal copy once remaining drivers migrate.

AI-generated, human reviewed

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed -- this duplication is intentional for phase 1 of #441 to keep this PR focused on removing the opendal dependency from QEMU without touching the other drivers.

For the follow-up, I think the cleanest approach is to have the opendal FlasherInterface inherit from this core one (they already have identical flash()/dump() signatures). That way any future contract change only needs to happen in one place. I'll track this in #441.


http_patch.assert_called_once_with(client=client, url="https://example.com/dump.bin", mode="wb")
local_patch.assert_not_called()
call_mock.assert_called_once_with("dump", "http_handle", None)
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[MEDIUM] No test coverage for _local_file_adapter streaming behavior

The tests cover path parsing and routing logic via mocks, but the actual _local_file_adapter streaming pipeline is never exercised directly. _AsyncIteratorStream and _FileWriteObjectStream contain non-trivial logic (lazy file open, chunked reading, progress attributes) that could break silently. Consider adding at least a focused unit test that exercises _local_file_adapter with a real temp file to verify the streaming and chunking behavior end-to-end.

AI-generated, human reviewed

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point. The streaming internals (_AsyncIteratorStream, _FileWriteObjectStream, chunked reading) are non-trivial and currently only tested indirectly through the routing mocks.

I'd prefer to add a focused streaming integration test as a follow-up rather than in this PR, since it would need to set up a mock DriverClient with a working resource_async pipeline, which adds meaningful complexity. The routing tests give us confidence the right code paths are selected, and the e2e QEMU tests exercise the full pipeline end-to-end. Will track this as a follow-up to #441.

For local files: streams via the existing resource_async mechanism using anyio.
For HTTP URLs: passes a PresignedRequestResource directly to the exporter,
which already handles presigned downloads via aiohttp.
"""
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[MEDIUM] Module-level docstrings may violate project comment policy

This file, driver/flasher.py, and flasher_test.py all contain module-level docstrings explaining the implementation approach. The project rules state comments should be a last resort when code cannot be self-explanatory. Existing comparable modules (e.g., opendal/client.py, opendal/driver.py) do not use module-level docstrings. The module names, class names, and method signatures already convey purpose -- consider removing these docstrings to stay consistent with the rest of the codebase.

AI-generated, human reviewed

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fair point -- removed all three module-level docstrings in 0c2d55c to match the rest of the codebase. The module/class/method names are self-explanatory.

pass

async def aclose(self):
pass
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[LOW] _AsyncIteratorStream.aclose() does not propagate close to wrapped async generator

The aclose() method is a no-op, so if the stream is abandoned before the async generator completes (e.g., on error), the file handle inside the generator may not be closed promptly. Adding await self.iterator.aclose() would ensure the async with block in the generator properly closes the file handle. Impact is minor (only affects error paths, GC will eventually clean up), but it would be cleaner.

async def aclose(self):
    await self.iterator.aclose()

AI-generated, human reviewed

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch. Fixed in 0c2d55c -- aclose() now calls await self.iterator.aclose() which will properly trigger the async with cleanup in the generator and close the file handle on error paths.


http_patch.assert_called_once_with(client=client, url="https://example.com/dump.bin", mode="wb")
local_patch.assert_not_called()
call_mock.assert_called_once_with("dump", "http_handle", None)
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[LOW] No test for dict-based multi-target flash

FlasherClient.flash() supports passing dict[str, PathBuf] for multi-target flashing (around lines 258-267 of flasher.py). No test covers this code path, including the ArgumentError case when both a dict and a target kwarg are provided. Adding a test for the happy path and the error case would improve confidence in this feature.

AI-generated, human reviewed

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed this would be good coverage. The dict-based multi-target flash path (including the ArgumentError guard) is straightforward but untested. I'll add tests for both the happy path and the error case as a follow-up -- keeping this PR focused on the core changes to avoid scope creep.


http_patch.assert_called_once_with(client=client, url="https://example.com/dump.bin", mode="wb")
local_patch.assert_not_called()
call_mock.assert_called_once_with("dump", "http_handle", None)
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[LOW] No test for dump with local path routing

TestFlasherClientRouting tests dump routing for HTTP URLs but has no corresponding test for dump routing to local file paths. Adding a test_dump_local_path case would round out the routing coverage.

AI-generated, human reviewed

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added test_dump_routes_local_path in 0c2d55c -- this mirrors the existing test_dump_routes_http_url test and verifies that local paths go through _local_file_adapter with the correct mode="wb" parameter.

@ambient-code
Copy link
Copy Markdown
Contributor Author

ambient-code bot commented Apr 13, 2026

Addressed @raballew's review feedback in 0c2d55c:

Fixed in this commit:

  • Removed module-level docstrings from flasher.py, driver/flasher.py, and flasher_test.py to match project conventions
  • Fixed _AsyncIteratorStream.aclose() to propagate close to the wrapped async generator (ensures file handle cleanup on error paths)
  • Added test_dump_routes_local_path to cover the missing dump routing test for local file paths

Tracked for follow-up in #441:

  • Having opendal FlasherInterface inherit from the core one (or removing it once remaining drivers migrate)
  • Adding focused streaming integration test for _local_file_adapter
  • Adding tests for dict-based multi-target flash (happy path + ArgumentError case)

Ambient Code Bot and others added 5 commits April 15, 2026 07:47
Move FlasherInterface to core jumpstarter package (jumpstarter.driver.flasher)
and create a simplified FlasherClient (jumpstarter.client.flasher) that streams
local files via resource_async and passes HTTP URLs as PresignedRequestResource
for exporter-side download, eliminating the need for the opendal library.

Update QEMU driver to import from the new core location and remove
jumpstarter-driver-opendal from its dependencies. Update tests to use
direct HTTP URLs instead of opendal Operator.

Closes #441

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The new FlasherClient in jumpstarter/client/flasher.py imports click
but it was not listed in the dependencies, causing import errors in tests.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Ensures the HTTP URL code path in FlasherClient is covered by tests,
since the integration test only exercises this path when no cached
image is available.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Remove unused `pathlib.Path` and `PresignedRequestResource` imports
that were flagged by ruff (F401).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…l test

- Remove module-level docstrings from flasher.py, driver/flasher.py, and
  flasher_test.py to match project conventions (per raballew review)
- Fix _AsyncIteratorStream.aclose() to propagate close to wrapped async
  generator, ensuring file handles are cleaned up on error paths
- Add test_dump_routes_local_path to cover dump routing for local file paths

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@ambient-code ambient-code bot force-pushed the ambient-fix/441-remove-opendal-qemu branch from 0c2d55c to 69cbb0e Compare April 15, 2026 07:47
Comment on lines +6 to +15
class FlasherInterface(metaclass=ABCMeta):
@classmethod
def client(cls) -> str:
return "jumpstarter.client.flasher.FlasherClient"

@abstractmethod
def flash(self, source, target: str | None = None): ...

@abstractmethod
def dump(self, target, partition: str | None = None): ...
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[MEDIUM] Still not addressed from previous review: The new core FlasherInterface and the existing jumpstarter_driver_opendal.driver.FlasherInterface are parallel ABCs with identical method signatures but no inheritance relationship. This means drivers switching from opendal's FlasherInterface to the core one (like QEMU did) are making a silent, potentially breaking change for any code doing isinstance checks. The opendal FlasherInterface should be updated to inherit from the core one, or at minimum re-export it, to establish a clear hierarchy.

AI-generated, human reviewed

Comment on lines +24 to +50
@dataclass(kw_only=True)
class _AsyncIteratorStream(ObjectStream[bytes]):
"""Wraps an async iterator as an ObjectStream for resource_async."""

iterator: Any
total: int | None = None

async def receive(self) -> bytes:
try:
return await self.iterator.__anext__()
except StopAsyncIteration:
raise EndOfStream from None

async def send(self, item: bytes):
raise BrokenResourceError("read-only stream")

async def send_eof(self):
pass

async def aclose(self):
await self.iterator.aclose()

@property
def extra_attributes(self) -> Mapping[Any, Callable[[], Any]]:
if self.total is not None and self.total > 0:
return {ProgressAttribute.total: lambda: float(self.total)}
return {}
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[MEDIUM] Still not addressed from previous review: _AsyncIteratorStream and _FileWriteObjectStream have no direct unit tests exercising their streaming behavior (receive/send/aclose lifecycle, error propagation on partial reads, EndOfStream on empty files, etc.). The existing routing tests mock these classes entirely, so actual I/O paths remain uncovered.

AI-generated, human reviewed


local_patch.assert_called_once()
http_patch.assert_not_called()
call_mock.assert_called_once_with("dump", "local_handle", None)
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[LOW] Still not addressed from previous review: No test for dict-based multi-target flash. FlasherClient.flash accepts a dict[str, PathBuf] and iterates over entries calling _flash_single for each, and also raises ArgumentError when both a dict and target are provided. Neither path is tested.

AI-generated, human reviewed

Comment on lines +234 to +237
if url is not None:
# HTTP URL: pass as presigned request for exporter-side download
with _http_url_adapter(client=self, url=url, mode="rb") as handle:
return self.call("flash", handle, target)
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[MEDIUM] The compression parameter is silently ignored when the path is an HTTP URL. _http_url_adapter does not accept a compression keyword, so _flash_single drops it without warning when routing to the HTTP path. The same applies to dump at lines 273-275. Either raise an error/warning when compression is specified with an HTTP URL, or pass it through to the exporter.

AI-generated, human reviewed

Comment on lines +270 to +278
"""Dump image from DUT"""
local_path, url = _parse_path(path)

if url is not None:
with _http_url_adapter(client=self, url=url, mode="wb") as handle:
return self.call("dump", handle, target)
else:
with _local_file_adapter(client=self, path=local_path, mode="wb", compression=compression) as handle:
return self.call("dump", handle, target)
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[LOW] Parameter naming is inconsistent between the driver-side FlasherInterface and the client-side FlasherClient. The driver's dump(self, target, partition) uses target for the destination resource and partition for the optional selector, while this client's dump(self, path, *, target) uses path for the destination and target for the selector. Since the client maps its target kwarg to the driver's partition positionally via self.call("dump", handle, target), this works at runtime but creates a confusing API surface. Consider aligning the naming.

AI-generated, human reviewed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants